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WAVELET-BASED IMAGE PROCESSING PATH 

FIELD OF THE INVENTION 

The present invention relates to the field of image processing; more 
5 particularly, the present invention relates to a wavelet-based image 
enhancement system. 

BACKGROUND OF THE INVENTION 

Image processing has many applications. One such application is in a 
1 0 digital copier. A digital copier typically has different modes for copying text 
or an image. The different modes, text or letter mode vs. photo mode, 
include special image processing techniques such as sharpening or 
smoothing of the scanned document, thinning of text, despeckeling, etc. 
Problems arise if the copied document has text and image content. In order 
1 5 to apply different techniques to text and image regions of a document, a 

segmentation of the document or a classification of the pixel of the scanned 
document is needed. Without out this classification, an oversmoothing of 
text in the photo-mode may occur and an enhancement of halftone noise in 
the text mode may occur. A consequence of this enhancement of halftone 
20 noise in the text mode can be the creation of severe Moire patterns after 
resolution reduction. Such artifacts are observed in many current digital 



copiers. A typical imaging path for digital gray scale copier is demonstrated 
in Figure 1. 

Referring to Figure 1, a scanned image 100 is input into processing 
block 101, which changes gray values in the scanned image to compensate 
5 for scarmer bias through lookup tables or other point operation (e.g., gamma 
correction). Next, different regions in the scanned image are classified (e.g., 
text, halftone, continuous tone, etc.) (processing block 102). The pixels are 
then filtered by processing block 103 depending on the classified pixel. Such 
filtering may include smoothing or sharpening of pixels based on their 

1 0 classification. After filtering, the pixels imdergo up and/ or downsampling 
by processing block 104. The pixels may also be subjected to resizing for 
enlarging/reducing. Next, the gray values of the pixels are changed 
through look up tables or other point operation (e.g., gamma correction) to 
compensate for printer bias (processing block 105). Once changing of the 

1 5 gray values has been done, halftoning is performed on the pixels (processing 
block 106). For non-copier applications, halftoning might be replaced with 
another output formatting step, the result of which is processed image 107. 

Single components of the digital imaging copier path are prior art, 
such as contrast enhancement of the scarmed image or halftoning for 



4 

printing. See U.S. Patent No. 5,883,973, entitled "Method and Apparatus for 
Processing a Document by Segmentation into Text and Image Areas," issued 
March 16, 1999 to Pasovici and Shu, and U.S. Patent No. 5,805,721, entitled 
"Method and Apparatus for Contrast Enhancement," issued September 8, 
5 1998 to Vuylsteke and Schoeters. There are also publications on image 
enhancement with wavelets that include denoising and contrast 
enhancement. For example, Zong, Laine, Geiser, and Wilson, "De-Noising 
and Contrast Enhancement via Wavelet Shrinkage and Non-linear Adaptive 
Gain," Proceedings of the SPIE, Vol. 2762, pgs. 566-574, 1996, Image 
1 0 enhancement using the Laplacian pyramid to add high frequency content to 
the image is also known. For example, see U.S. Patent No. 5,717,789, entitled 
"Image Enhancement by Non-linear Extrapolation in Frequency Space," 
issued February 1998 to Anderson and Greenspan. 



5 

SUMMARY OF THE INVENTION 

A system is described. In one embodiment, the system comprises an 
image processing path and a print engine coupled to the image processing 
path. The image processing path processes an input image in a wavelet 
5 domain. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention will be imderstood more fully from the 
detailed description given below and from the accompanying drawings of 
various embodiments of the invention, which, however, should not be taken 
5 to limit the invention to the specific embodiments, but are for explanation 
and understanding only. 

Figure 1 illustrates a prior art image processing path for a grayscale 

copier. 

10 

Figure 2A is a block diagram of one embodiment of a system having a 
wavelet-based image processing path. 

Figure 2B is a block diagram of one embodiment of a digital copier. 

15 

Figure 2C is a block diagram of one embodiment of a wavelet-based 
processing path for a digital copier. 

Figure 3 illustrates a classifier for use in a copier or printer. 

20 

Figure 4A illustrates the formation of wavelet coefficients of a 
halftone and diagonal line pattern using a 2D Haar wavelet transform . 



7 

Figure 4B illustrates a schematic overview of the classification based 
on the first level wavelet coefficients. 

Figure 5 illustrates 3x3 windows in the LH, HL, HH bands at level 
5 one of the wavelet decomposition. 

Figure 6 illustrates exemplary coefficient templates. 

Figure 7 is a block diagram of one embodiment of a copier. 

10 

Figure 8 is a block diagram of one embodiment of a computer system. 



8 



DETAILED DESCRIPTION OF THE PRESENT INVENTION 

A method for processing an image and apparatus for doing the same 
are described. In the following description, numerous details are set forth in 
order to provide a thorough understanding of the invention. It will be 

5 apparent, however, to one skilled in the art, that the present invention may 
be practiced without these specific details. In other instances, well-known 
structures and devices are shown in block diagram form, rather than in 
detail, in order to avoid obscuring the present invention. 

Some portions of the detailed descriptions which follow are presented 

10 in terms of algorithms and symbolic representations of operations on data 
bits within a computer memory. These algorithmic descriptions and 
representations are the means used by those skilled in the data processing 
arts to most effectively convey the substance of their work to others skilled 
in the art. An algorithm is here, and generally, conceived to be a self- 

1 5 consistent sequence of steps leading to a desired result. The steps are those 
requiring physical manipulations of physical quantities. Usually, though 
not necessarily, these quantities take the form of electrical or magnetic 
signals capable of being stored, transferred, combined, compared, and 
otherwise manipulated. It has proven convenient at times, principally for 



reasons of common usage, to refer to these signals as bits, values, elements, 
symbols, characters, terms, numbers, or the like. 

It should be borne in mind, however, that all of these and similar 
terms are to be associated with the appropriate physical quantities and are 
5 merely convenient labels applied to these quantities. Unless specifically 

stated otherwise as apparent from the following discussion, it is appreciated 
that throughout the description, discussions utilizing terms such as 
"processing" or "computing" or "calculating" or "determining" or 
"displaying" or the like, refer to the action and processes of a computer 

1 0 system, or similar electronic computing device, that manipulates and 

transforms data represented as physical (electronic) quantities within the 
computer system's registers and memories into other data similarly 
represented as physical quantities within the computer system memories or 
registers or other such information storage, transmission or display devices. 

1 5 The present invention also relates to apparatus for performing the 

operations herein. This apparatus may be specially constructed for the 
required purposes, or it may comprise a general purpose computer 
selectively activated or reconfigured by a computer program stored in the 
computer. Such a computer program may be stored in a computer readable 
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storage medium, such as, but is not limited to, any type of disk including 
floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only 
memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, 
magnetic or optical cards, or any type of media suitable for storing electronic 
5 instructions, and each coupled to a computer system bus. 

The algorithms and displays presented herein are not inherently 
related to any particular computer or other apparatus. Various general 
purpose systems may be used with programs in accordance with the 
teachings herein, or it may prove convenient to construct more specialized 

1 0 apparatus to perform the required method steps. The required structure for 
a variety of these systems will appear from the description below. In 
addition, the present invention is not described with reference to any 
particular programming language. It will be appreciated that a variety of 
programming languages may be used to implement the teachings of the 

1 5 invention as described herein. 

A machine-readable medium includes any mechanism for storing or 
transmitting information in a form readable by a machine (e.g., a computer). 
For example, a machine-readable medium includes read only memory 
("ROM"); random access memory ("RAM"); magnetic disk storage media; 



11 

optical storage media; flash memory devices; electrical, optical, acoustical or 
other form of propagated signals (e.g., carrier waves, infrared signals, digital 
signals, etc.); etc. 

5 Overview 

The processing described herein provides for image processing 
between scanned (or otherwise sensed) digital images and digital images for 
display or printing. This image processing might be in a digital image in, 
digital image out device such as, for example, a printer controller (e.g., for a 

1 0 network) or performed in a general purpose computer. The image 

processing described herein might be combined with an analog input device 
in, for example, a scanner or digital camera, with a print engine in a printer 
or with another output device, or with both an input and output device, for 
example in a digital copier, facsimile machine or multifunction machine. 

1 5 The wavelet-based image processing described herein may be used to 

compensate for differences between input and output systems. Some image 
in and image out systems have matched input and output devices that 
require little or no image processing between input and output for particular 
types of images. For example, some video cameras and video displays are 
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well matched and good results for images of natural scenes require little or 
no intermediate processing. In contrast, some systems have dissimilar input 
and output devices or complex/compoimd image types where significant 
processing is required. For example, an input device might require less 
5 noise, higher sharpness and/or a different size/scale than the input device 
provides. Additionally different images or different positions in the same 
image might have different enhancement. The wavelet-based image 
processing described herein may be used to process an image from an input 
system (e.g., device) so that it "matches" the output system (e.g., device). 

10 

Overview of a Wavelet-based Image Processing System 

A wavelet-based imaging path for a digital copier or multifunction 
machine having automatic adaption of image processing routines to various 
characteristics in the image such as, for example, text and backgroimd is 
1 5 described. 

The wavelet-based image processing may occur after non-wavelet- 
based processing blocks or modules and/ or before other non-wavelet-based 
processing blocks or modules. Figure 2A is a block diagram of a system 
having a wavelet-based image processing path. Referring to Figure 2A, an 
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image source 250 generates image data. Image source 250 may comprise, for 
example, a digital camera, scaraier, a network or charmel, storage, etc. Then 
processing logic, optionally, changes gray values through look up tables 
(e.g., gamma correction) (processing block 251). Alternatively, the image 

5 data may be subjected to a point operation. Afterwards, processing image 
data is received by wavelet-based image processing block 252 for wavelet- 
based processing. After wavelet based processing, processing logic 
optionally changes the gray values of the image data through look up tables 
(e.g., gamma correction) or, alternatively, a point operation. Next, 

1 0 processing logic performs halftoning on the image data or some other form 
of preparation for output (processing block 254) and delivers the processed 
image data to an image distinction 255. Image distinction 255 may comprise, 
for example, a printer, print engine, display, network or channel, or storage. 
Figure 2B is a block diagram of a copier. Referring to Figure 2B, 

1 5 processing logic receives a scarmed image 280, and then, optionally, changes 
gray values in the scanned image 200 through look up tables (e.g., gamma 
correction) (processing block 281). Alternatively, scarmed image 280 may be 
subjected to a point operation. Afterwards, processed image data is received 
by wavelet-based image processing block 252 for wavelet-based processing. 
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After wavelet-based processing, processing logic optionally changes the 
gray value of the image data through look up tables (e.g., gamma correction) 
or, alternatively, a point operation. Next processing logic performs 
halftoning on the image data or some other form of preparation for output 

5 (processing block 254), thereafter outputting processed image 285. 

Figure 2C is a block diagram of one embodiment of a wavelet-based 
image processing path for a digital copier. In one embodiment, this image 
processing path replaces processing blocks 102-106 in Figure 1 ia a copier or 
other image processing device. The processing may be performed by 

1 0 processing logic that may comprise hardware, software or a combination of 
both. 

Referring to Figure 2C, forward wavelet transforms block 201 
performs a wavelet transform on the scarmed image to create a set of 
coefficients. The coefficients may be optionally sent to wavelet-based 
1 5 classifier processing block 202A, which classifies the coefficients. 

Alternatively, image data may be directly input to a non-wavelet-based 
classifier block 202B. The classifier information output from classifier 
processing blocks 202A and 202B may be used in cooperation with 



processing block 203-206, 209, 210 and 202C. These processing blocks are 
described below. 

Processing logic performs thresholding on the coefficients output 
from wavelet transform block 201 (processing block 203). After 

5 thresholding, processing logic rescales coefficients (processing block 204). 
Next, processing logic filters coefficients (processing block 205). 

After filtering coefficients, processing logic may optionally perform 
upsampling or downsampling (processing block 206). Subsequently, 
processing logic performs an inverse wavelet transform on the coefficients 

1 0 (processing block 207). In an alternative embodiment, processing logic 
performs an inverse wavelet transform that includes upsampling and /or 
downsampling (processing block 208). Note that if processing logic 
performs an inverse wavelet transform in combination with 
up /downsampling (i.e., processing block 208), the processing block 206 is 

15 not performed. 

After performing an inverse transform, processing logic changes gray 
values on the image data through a look up table or other point operation 
(e.g., gamma correction) (processing block 209). Processing logic then 
performs halftoning on the image data (processing block 210). The output of 
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halftoning block 210 is a processed image 220. Note that processing blocks 
209 and 210 are not performed in the wavelet domain. 

The use of signal-adaptive nature of wavelet domain image 
processing, including a description of each of these blocks follows. 

5 

Critically Sampled or Overcomplete Discrete Wavelet Decomposition 

Initially, in the image processing path, forward wavelet transform 
block 201 performs a wavelet decomposition on the scanned image up to a 
chosen level L. L may be any number of levels, but 2 or 3 levels is sufficient 

1 0 for many applications. This transform may be critically sampled or 
overcomplete. Besides overcomplete (fully redundant) and critically 
sampled wavelet transforms, a transform with reduced redundancy or a 
complex transform could also be used. An example for reduced redundancy 
is the computation of all phases of the critically sampled wavelet transform 

15 at the first level of resolution and only some of the phases, such as every 
other phase at the second level of resolution (see Selesnik, I. W. ''Smooth 
Wavelet tight Frames with Limited Redimdancy," Technical report. Dept. 
Electrical Engineering, Polytechnic University, Brooklyn, NY, August 1999). 
Alternatively, the phases could be chosen dynamically, e.g., by selecting 
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those that contain the most energy. For example, see Coiftnan, R. R., 
Donoho, D. L. 'Translation-invariant De-noising/' in Wavelets and Statistics 
(Editor A. Antoniades), Springer Lecture Notes, Springer-Verlag, 1995. 
Complex transforms compute real and imaginary parts wavelet coefficients, 
5 which results in a redundancy factor of two. The imaginary parts contain 
phase information on an image region. For example, see Kingsbury, N. "The 
dual-tree complex wavelet transform: a new efficient tool for image res- 
toration and enhancement,'' Proceedings European Signal Processing Conf 
1998, pp, 319-322. 

10 If the image is given by NxN data points then, when using a critically 

sampled wavelet transform for L levels, the decomposition is also given by 
NxN data points, namely wavelet and scaling coefficients. When using the 
overcomplete wavelet transform for L levels, the decomposition is given by 
(3L+l)NxN data points. 

15 In both cases, wavelet coefficients dj for levels 1=1 to L and scaling 

coefficients c^ at the coarsest level have to be stored. In one embodiment, an 
overcomplete Haar wavelet transform is used. An example of a ID critically 
sampled Haar transform (without L^- normalization, low pass = a + b, high 
pass = a - b; ignore boundaries) is given below: 
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Table 1 
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An example of overcomplete Haar transform is as follows: 



Table 2 
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Examples of filter coefficients for different wavelet systems are as follows 
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Haar: - 1] normalized by 1/ ^f2. 

2-6: [ill [-1/8 -1/8 1 -1 1/8 l/S] normalized by 1/V2. 

5_3: [_i/4 1/2 3/2 1/2 -l/4l[l/4 -1/2 I/4] normalized by 1/V2. 

2-10: [1 ll[3 3-22 -22 128 -128 22 22 -3 - S]/ 128 normalized by 1/V2. 

9-7: [0.037828 -0.023849 -0.110624 0.377402 0.852699 0.377402 

-0.110624 - 0.023849 0.037828], [0.064539 - 0.040689 - 0.418092 

0.788486 - 0.418092 - 0.040689 0.064539] 
daub6: [0.0352 -0.0854 -0.1350 0.4599 0.8069 0.3327], [-0.3327 0.8069 

-0.4599 -0.1350 0.0854 0.0352]. 

The forward low pass and high pass filters are specified. For 
symmetric biorthogonal filters, the inverse filters are the forward filters with 
alternating signs changed starting with the second and reversed. For 
example for the 2-6 filter: 




Forward high pass 



Inverse low pass 




This selection of filters covers the characteristics for the most widely 
used wavelet families (orthogonal, biorthogonal, symmetric, even-length. 
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odd-length). The transform may also be any LeGaul-Tabatabai biorthogonal 
transform. 

For a two dimensional wavelet transform, both low pass and high 
pass filters are applied horizontally and vertically. The four resulting 
5 subbands are: 

Subband horizontal filter vertical filter 

LL low pass low pass 

HL high pass low pass 

LH low pass high pass 

10 HH high pass high pass 

Multiscale Sharpening, Smoothing and Denoising 

In one embodiment, thresholding and multiplication are performed 
15 on the wavelet coefficients to achieve denoising, sharpening, and smoothing. 
A criterion dictates whether thresholding or multiplication are applied. In 
one embodiment, the criterion comprises the magnitude of wavelet 
coefficients. Alternatively, a classifier could be used to indicate whether to 
apply thresholding and/ or multiplication. 

20 



Denoising by Thresholding 

Denoising may be performed by setting all coefficients below a given 
threshold to zero. This technique is known as hard-thresholding and is well- 
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known in the art. As an alternative, soft-thresholding, e.g., shrinking the 
coefficients above the threshold toward zeros by the amoimt of the threshold 
can also be considered. Such a technique may be advantageously used for 
critically sampled and overcomplete wavelet decompositions, and is 

5 described in Donoho, ''Denoising by Soft-Thresholding", IEEE Transactions 
on Information Theory, 41(3): 613-627, 1995. The threshold may be one that 
is calculated for the presence of additive Gaussian White noise in an image. 
This threshold depends only on the size of the data and the variance of the 
noise which is estimated from the wavelet coefficients using a well-known 

1 0 procedure (e.g., statistical estimation of variance). Other choices of 

threshold selection may be used. These may be calculated from a series of 
test images. 

There are additional strategies for thresholding available. There may 
be a global threshold or different thresholds for different levels of 
1 5 decomposition or different bands (LH, HL, HH). The thresholds may be set 
manually. Thresholds may depend on one or more of the following 
parameters: 

1. Standard deviation or median of absolute values of coefficients in one 
region (e.g., HH coefficients at level 1) (global threshold) 
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2. Standard deviation or median of absolute values of HH coefficients at 
each level (level-dependent threshold) 

3. Standard deviation or median of absolute values of LH/HL/HH 
coefficients at first level (band-dependent threshold). For the example 
of two levels of decomposition three thresholds are chosen 

4. Standard deviation or median of absolute values of LH/HL/HH 
coefficients at each level (level-band-dependent threshold). For the 
example of two levels of decomposition, six thresholds are chosen. 

5. Manually set thresholds (global, level-dependent, level-band 
dependent) using controls that are part of the user interface of the 
device. 

6. Local thresholding by computing local variances or medians of 
coefficients in between a band. 

7. Given classifier: select for each strategy (global, level-dependent, 
level-band dependent) a number of thresholds corresponding to the 
number of classes. Then use those classifier-dependent thresholds. 
Laplacian based techniques applied to wavelets do not perform the 

superior wavelet-only denoising. It is important to note that wavelet 
denoising techniques do not apply to Laplacian pyramid decompositions. 
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Laplacian pyramid decompositions can be used only for systems where 
inferior or no denoising is acceptable. 

Rescaling of Coefficients 

5 After any denoising is completed, the wavelet coefficients that are 

non-zero are modified for the purpose of sharpening or smoothing. 
In one embodiment, sharpening or smoothing of the image is obtained by 
multiplication of wavelet coefficients with a level-dependent parameter ji^ > 
0. For sharpening, the high frequency bands should be multiplied with a 

1 0 larger factor than the low frequency bands, whereas for smoothing the low 
frequency bands should be multiplied with larger factors than the high 
frequency bands. One way to obtain this rescaling is to multiply the 
coefficients at level j with |ij = R • 2"^ The parameter R functions as a 
renormalization factor for preserving the global energy of coefficients and a 

1 5 is the parameter that determines the degree of sharpening (a < 0) or 

smoothing (a > 0). For more information, see U.S. Patent Application Serial 
No. 09/467,544, entitled "Multiscale Sharpening and Smoothing With 
Wavelets," filed December 10, 1999, assigned to the corporate assignee and 
incorporated herein by reference. There are many choices for the parameter 
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R that depend on how the energy of an image is measured. One way woiild 



be to preserve the norm ^XK.j| of the image before and after rescaling, 
i. e. compute R with 



5 If the scanned image is assumed to have a Hoelder regularity of y (typically 
Y ^ 1 ), an approximation of can be obtained by the following estimate. 



2{L+l)2(Y+a) _22(Y+a)' 
22(Y+a) _ J 



This approximation can be computed a-priori and does not depend on the 
1 0 actual values of wavelet coefficients. 

Instead of using the norm, there are other choices for the norm. 
Using the theory of Besov space, it is possible to preserve a specific Besov 
norm of the image. This norm is defined as a weighted sum of wavelet 
coefficients. The parameter R can be computed as 



15 R'^ = 



















1 








I ^ ) 
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for 0 < p,q < oo. 

For p,q=^/ the term (^l-f J' m the previous equation is substituted by 
sup\djj^\^ 

In one embodiment, the choice of the parameters a and R can be 
5 controlled by the user interface of a device, giving the user the opportimity 
to select more /less sharpening, or more /less smoothing. This 
parameterization of sharpening and smoothing allows a fine graduation of 
degrees of enhancement* 

As shown in Figure 2 information obtained from a classifier can be 
1 0 used to determine different parameter a^, for different classified areas 
in the image. 

Filtering of Coefficients 

The frequency content of a single subband at a specific level of 
decomposition can be modified by a filtering of that subband. For example, 
1 5 additional sharpening of the image could be performed by filtering the LL 
component at the largest level of decomposition with a sharpening filter. In 
a similar way, additional smoothing could be performed by applying a 
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smoothing filter to the LL component. Another possibility is to remove 
periodic patterns in a specific band by applying a special filter (e.g., a 
bandpass or notch filter) to that specific band /component of the wavelet 
decomposition. Thus, a formulation filter is applied in the wavelet domain 
5 (e.g., the LL domain) that is equivalent to a larger filter in the pixel domain. 

A filter may be used that is suited to reduce or remove special 
periodicities out of the image. The output of this filter may be used to select 
different denoising thresholds for halftone and text regions. Periodicity 
detection can be done with autocorrelation or similar fimctions. For 
1 0 example. 



or 



15 . 

There is periodicity with period 2 at location i if 
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There is periodicity with period 3 at location i if 

// <fi 



III some applications, the period (of the periodicity) is known. The 
period might be determined from the screen size of a halftoned original to 
be copied and the resolution of the copier's scanner. For example, consider a 
period of 3. Small amplitude wavelet coefficients might be filtered in one 
dimension by 

^ filtered ^ + + ^f+l ) 

3 

For the overcomplete Haar at level 1 (and no adaptive processing 
such as denoising), this is equivalent to the linear filter [12 21] in the spatial 
domain. 

A two dimensional example is 



For the overcomplete Haar at level 1 this is equivalent to the 
following linear filter in the pixel domain. 
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"12 2 1" 
2 4 4 2 
2 4 4 2 
1 2 2 1_ 

Similar 1x5 or 5x5 filters might be used for period 5. When even 
symmetric wavelet filters are used, even period periodicities are removed 
easily by the wavelet and extra processing is typically only desired for odd 
5 periods. 



Use of a Classifier and an Exemplary Classifier 

In one embodiment, denoising, sharpening and smoothing of wavelet 
coefficients is controlled using a classifier. A classifier could be computed 
1 0 based on information extracted from the wavelet coefficients or from some 
other information source, e.g. the image pixels (Figure 2). 

As a decision making step classification is part of a segmentation 
process that decomposes an image into various components. 

Classification or segmentation can be used for many things in a 
1 5 system. It can be used to control one or more of: de-noising, 

sharpening/smoothing, upsampling/downsampling, change of gray values 
through lookup table (e.g. gamma correction), halftoning, imder-color 
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removal and/or image compression. Non-wavelet based filtering methods 
may have a single filter that performs denoising and sharpening /smoothing, 
and classification would select between different filters. With wavelet-based 
processing, not only are different parameters used for denoising and 

5 sharpening/ smoothing, but also different scales or different wavelet 

subband may use different parameters. This flexibility is an advantage of 
wavelet based processing. Having classification only control desired 
parameters reduces the effect of classification errors. For example for de- 
noising and sharpening/smoothing in one system, perhaps only de-noising 

1 0 is controlled by classification or denoising at the first transform level. In 
another system, perhaps denoising and sharpening/ smoothing at the first 
two transform levels are affected by classification. 

Regardless of the source of the classification, the result can be used 
for a variety of things including: setting smoothing parameters, setting 

1 5 sharpening parameters, choosing an "ordinary" linear filter for sharpening 
or smoothing, controlling the under color removal process in color 
copiers/printers, as an input to a document classification algorithm, as an 
input to a compression algorithm (like JBIG 2), as a way to control a 
halftoning algorithm (or to turn off halftoning for certain image types switch 
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between error diffusion and ordered dither). For example, when performing 

imder color removal, one classifier may be as follows: 

classify as text or line -4 black = min (cyan, magenta, yellow) 

classify as not text or line black = a min (cyan, magenta, 
yellow) where 0<a<l, for example a =0,75 

In a compression framework, JBIG2 and MRC use classification in the 
sense that the encoder must decide how to classify, but no explicit 
classification scheme is specified (ISO/IEC 16485: Information technology - 
Mixed Raster Content (MRC)). These methods could use a wavelet-based 
classifier in the encoder. Such a classifier is shown in Figures 3A-3C. 

Classifiers can distinguish between a variety of different image area 
types : 

• text and line art versus other 

• black and white text versus colored or gray text 

• text with patterned backgrovmds versus text on plain backgrounds 

• text of various sizes 

• halftone versus other 

• gathered dot halftones versus other halftones 

• gathered dot halftones of various screen frequencies 

• continuous-tone image versus other 
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Spatial domain classifiers can be mapped to the wavelet domain. 
Each wavelet coefficient has a support in the spatial domain. The AND, OR 
or majority of the spatial domain classifications for the area of support for a 
coefficient can be used for the classification of a coefficient. Similarly, a 
5 wavelet domain classifier can be mapped to the spatial domain. The AND, 
OR or majority of the wavelet domain classifications for coefficients that 
have support including a particular pixel can be used for the classification of 
a pixel. 

Many kinds of classifiers can be implemented in the wavelet domain. 
1 0 Classifiers can be based on the value of a coefficients (Figure 6A) or on 
statistics from regions of coefficients (Figure 6B,C,D), a more thorough 
explanation of which are given below. Possible statistics include: 
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Mean x = (l/n)^x and standard deviation sd = 



• mean and first absolute moment — ^ 

• minimum and maximum 



x-x 



• mmimum 



• maximum 
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A location might be classified as text (and not halftone or continuous 
tone image) if the mean of region was about a threshold Tl and if the 
variation (standard deviation or first absolute moment) was below a second 
threshold T2. 

5 In one embodiment, to compute a classifier from wavelet coefficients, 

the local standard deviations of wavelet coefficients in different bands, levels 
or across bands and levels, may be computed. For example, if each level has 
three bands (LH, HL, and HH), then the mean for a detail coefficient is: 

dm+dHL+dHH 

ew— ~~ 

3 

1 0 The standard deviation across the bands is 

Depending on those measures, text halftone and background regions can be 
differentiated and a different denoising, smoothing, and sharpening 
parameter(s) may be applied. For example, text may be identified when the 
1 5 standard deviation of coefficients at one or over several levels is small, and 
halftone may be identified if the standard deviation is high. This makes it 
possible to distinguish halftone portions from lines. Thus, different 
denoising, sharpening or smoothing parameters may be selected for spatial 
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areas (e.g., local regions in wavelet components), different bands, 
resolutions. 

In one embodiment, a halftone-vs-line classifier is used in a wavelet- 
based image processing path of a copier. The difference between halftoned 

5 areas and text areas is that halftone areas consist of periodic patterns, 
whereas text consists of constant gray values in one direction. 

A vertical line in the image produces large wavelet coefficients of the 
same sign along a vertical line in the HL band, whereas the coefficients next 
to the line are small. A horizontal line produces large wavelet coefficients of 

1 0 the same sign along a horizontal line in the LH band. A diagonal line 

produces middle sized wavelet coefficients of the same sign along a diagonal 
line in the HL, LH and HH band. See Figure 4A. 

On the first level of decomposition, a halftoned area produces large 
coefficients of positive and negative sign in the HL and LH band. These 

1 5 coefficients are not connected, but rather "uniformly distributed''. In the 
HH bands, a halftone area produces non-connected large size coefficients. 
Figure 4 A illustrates the formation of wavelet coefficients of a halftone and 
diagonal line pattern using a 2D Haar wavelet transform. 
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The difference in formation of wavelet coefficients of lines and 

halftone areas can be captured by computing the mean and the standard 

deviation in a window of a special length directed in horizontal, vertical and 

both diagonal directions. In terms of those parameters the characterization 

5 of lines and halftone areas is shown in Table 3. 

Table 3 - Characterization of lines and halftone area in directional window in 
the wavelet domain. 





LH: mean 


LH: std 


HL: mean 


HL: std 


HH: mean 


HH: std 


horizontal lines 


large 


small 


small 


small 


small 


small 


vertical lines 


small 


small 


large 


small 


small 


small 


diagonal lines 


medium 


small 


medium 


small 


medium 


small 


halftone 


medium 


large 


medium 


large 


medium 


large 



1 0 It becomes apparent that the mean computed in a horizontal or 

vertical window is a sufficient criterion for separating horizontal and 
vertical lines from halftone or diagonal lines. For distinguishing diagonal 
lines from halftone, the standard deviation has to be computed as an 
additional criterion. This classification also works on a level larger than one, 

1 5 if the lines have a specific thickness and the halftone a specific frequency. 
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However, since this is information in general not available a priory, the 
classification is performed only at the first level of decomposition. 

In one embodiment, different classification thresholds {T^,T^,T:^,T^) are 
chosen to classify line and halftone areas in HL/ LH and HH bands. An 
5 example for a window of length 3 is: 



if (mean of horizontal window centered in LH-coefficient d(i/j) > AND 

std of horizontal window around coefficient d(i,j) < T^) 
then classify 3x3 window around (i,j) as being horizontal line area 

1 0 else if (mean of vertical window centered in HL-coefficient d(i,j) > T, AND 
std of vertical window around coefficient d(i,j) < T2) ) 
then classify 3x3 window aroimd (i,j) as being vertical line area 

else if (mean of diagonal SW-NE window centered in HH-coefficient d(i,j) > T3 
15 AND 

std of diagonal SW-NE window aroimd coefficient d(i,j) < TJ 
then classify 3x3 window around (i,j) as being diagonal line area 

else if (mean of diagonal NW-SE window centered in HH-coefficient d(i,j) > T3 
20 AND 

std of diagonal NW-SE window aroimd coefficient d(i,j) < TJ 
then classify 3x3 window aroimd (i,j) as being diagonal line area 

else 

25 classify 3x3 window around (i,j) as being halftone area 

A schematic overview of the classification based on the first level 
wavelet coefficients is shown in Figure 4B. This classification can work 
successfully in the line and halftone area. 
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Figure 5 illustrates 3x3 windows in the LH, HL, HH bands at level 
one of the wavelet decomposition. The bold coefficients are those used for 
the classifier. 

In one embodiment, for construction of the classifier, the following 
steps are followed. 

1 . Compute mean of the three bold values in each box in Figure 5, 
i.e. 

mean_LH = (x_4+x_5+x_6)/3, 
mean_HL = (x_2+x_5+x_8)/3, 
meanl.HH = (x_7+x_5+x_6)/3, 
mean2_HH = (x_l+x_5+x_9)/3. 

2. Compute standard deviation of the three bold values in each 
box, i.e. 

std_LH = sqrt([(x_4-mean_LH)^2 + (x_5-mean_LH)^2 + (x_6- 
mean_LH)^2]/3) 

std_HL = sqrt([(x_2-mean_HL)^2 + (x_4-mean_HL)^2 + (x_8- 
mean_LH)^2]/3) 

stdl_HH = sqrt([(x_7-meanl_HH)^2 + (x_5-meanl_HH)^2 + 
(x_3-meanl_HH)^2]/3) 

std2_HH = sqrt([(x_l-mean2_HH)^2 + (x_5-mean2_HH)^2 + 
(x_9-mean2_HH)^2] /3). 

3. Set the complete 3x3 window in the classifier image (positions 
of x_l,..x_9) to white 

if (abs(mean_LH) > thresh_l && std_LH < thresh_2) OR 
if (abs(mean_HL) > thresh_l && std_HL < thresh_2) OR 
if (abs(meanl_HH) > thresh_3 && stdl_HH < thresh_4) OR 
if (abs(mean2_HH) > thresh_3 && std2_HH < thresh_4). 
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This version of the classifier uses the mathematical definition of 
standard deviation. Of course, the sqrt can be eliminated by adjusting the 
thresholds thresh_2 and thresh_4. There may also be the possibility of 
substituting the squares {.)^2 by absolute values 1 . 1 . Then the thresholds 
5 thresh_2 and thresh_4 will have to be chosen appropriately. 

A wavelet-based classifier could also be used for nonwavelet-based 
operations such as halftoning or gamma correction. One example is to 
detect text of particular sizes in the wavelet domain and to use OCR (optical 
character recognition) on the text in the pixel domain, perhaps with 
1 0 parameters chosen based on the detected font size. 

Wavelet coefficients from different decomposition levels can be used 
in a classifier. 

Classifications can also effect a single coefficient or can effect a region 
of coefficients. 

1 5 In one embodiment, a wavelet coefficient is classified as being in a 

text region if the statistics for it or any of its eight neighbors meet thresholds 
for text. 

Similar to the first method, in an alternative embodiment, a region 
around that pixel is classified instead of a single pixel This is essentially 
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looking at the effect of one pixel on a region. The size of the region may 
depend on the scale L (e.g., the regions are selected to be of size 2^L x 2^L). 

In yet another embodiment, for each scale l=li...L, a threshold T. is 
selected. A pixel at position (i,j) is classified as text only if the coefficients at 
the corresponding positions at all scales I =1-1..L are the corresponding 
threshold. Otherwise, the pixel is classified as backgroimd. 

In still another embodiment, and similar to the previous method, a 
complete region aroimd the located pixel is classified as text, as opposed to 
single pixels. Possible size of the regions include, but are not limited to: 2^L 
x2^L. 

In one embodiment, the classification of an image into text, half-tone, 
continuous tone (or other types) is performed in the wavelet domain or in 
the spatial domain by a variety of techniques. 

Resolution Reduction and Magnification 

For resampling of an image (reduction or enlargement), an 
interpolation filter may be used. Filters with different characteristics lead to 
different characteristics in the resampled image. Common problems after 
resampling are increased moire artifacts, less crisp text as well as broken text 



39 

and lines. One reason for the occurrence of those artifacts is that the global 
filter characteristics of the interpolation filter do not match the 
characteristics of a local region in the image. For reduction of moire artifacts, 
a filter with random elements would be well suited to destroy the periodic 
5 noise. For downsampling in text regions, a sharpening filter may be used. 
For upsampling of lines, a smoothing filter may be used. 

Downsampling Combined With Inverse Transform 

Since resampling with a linear interpolation filter is a linear operator, 
it can be combined with the linear wavelet transform into one operator. 

1 0 Therefore, the resampling can be combined with the inverse wavelet 

transform into one filtering step, and resampling can be done directly on the 
wavelet coefficients. Since wavelet coefficients contain information on 
smooth and non-smooth parts of an image, different interpolation filters can 
be chosen depending on the absolute magnitude of wavelet coefficients or a 

1 5 classifier output or on other wavelet-based criteria or without information 
from an external classifier. 

For example, for downsampling of a one-dimei\sional signal with a 
length-4 interpolation filter [a^^a^a^Aal (e.g.. Keys filter), given a Haar 
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wavelet decomposition, the interpolation filter can be implemented directly 
on the two finest level of the decomposition. Out of the redimdant Haar 
wavelet coefficients, select the critically sampled coefficients with phase such 
that the support of the coefficients is in the length four window. In detail, in 
order to compute an interpolation point between x^i and x^.^^ (length-4 filter 
applied to window x^.^to x^J, the wavelet coefficients d],dl^a.t level 1 and 
d- at level 2 and the scaling coefficient sf at scale 2 have to be used. If the 
inverse transform step for reconstructing the sample x^, is sf + df 4- d] +dl^, 
then the exact interpolation filtering procedure on wavelet coefficients is 

xjnt = (ao + + + a3 >f + (ao + ai - - )df + (ao - )d| + (a^ - )dl, 

For 4-tap linear filters that preserve the DC value, i.e. aQ-^ei^-^a.^-i-^^ = 1, the 
above equation reduces to 

x_int = sf + (ao -h ai - a^ - ei^ )df + (ao " )d| + (a^ - a3 )dl, 

Different values for ei^^a^Ay^ yield different Keys filters. One possible 
way of performing a signal-adaptive dowrisampling in the wavelet domain 
is, for a given a threshold T for wavelet coefficients, select a Key filter that is 
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suited for text if the wavelet coefficients at level two \d^\ is larger than T and 



select a different Keys filter if di | <T. 

In one embodiment, thresholds for selecting the interpolation method 
or filter may vary with the amount of upsampling and for downsampling. 
5 For example, if a particular amount of dowrisampling is known to be 

performed, an interpolation method or filter may be selected that is different 
than that which would be used if no such amoxmt or a different amount of 
downsampling was being performed. 

Similarly, the thresholds for both denoising and interpolation may 
1 0 vary with the amount of upsampling and/or downsampling. The 

thresholds for classification and interpolation may vary with the amount of 
upsampling and/ or downsampling. Similarly, the windows upon which 
they operate may vary with the amount of upsampling and/ or 
downsampling. 

1 5 In one embodiment, the filter varies with the amount of up and/or 

downsampling, the periodicy filtered varies with the amoimt of up and/ or 
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downsampling, and /or the filter support varies with the amount of up 
and /or downsampling 

For a one-dimensional signal of length N, choosing matrix notation, 
the linear interpolation transform is represented by matrix Fj and the 
forward wavelet transform by the matrix W^. The matrix representing the 
combined interpolation and the inverse transform W^'Ms (Fj W^'^). 

In general, given the interpolation filter and the wavelet transform, it 
is possible to store the filter coefficients for the combined interpolation- 
inverse transform in a look-up table and use that table for the filtering. If the 
interpolation filter is of length Np and the maximum of the length of the 
wavelet transform analysis lowpass and highpass filter is then the 
regular filter bank algorithm for the inverse transform can be used from 

level L down to level L*^ where 



= logliV^,^ J+ 1) with N^^p = L. 2fr(^ w 1)/ 
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For the remaining levels L* to 1 the modified filtering procedure represented 
by the matrix Fj W^:' has to be applied. It can be also implemented at all 
levels 1<L*. 

A possible application for downsampling in a multifimction 
5 machine /copier /etc. is to store the combined interpolation-inverse wavelet 
transform coefficients for a fixed transform and a family of interpolation 
filters that covers a range of sharpening and smoothing filters. If a wavelet 
coefficient at level L or L*is above a given threshold (e.g., belonging to a 
strong text-edge), then the coefficients for a sharpening filter + inverse 
1 0 transform filtering is chosen. If the coefficient is below the threshold (e.g., in 
backgroimd regions), then the coefficients for a smoothing filter + inverse 
transform filtering is chosen. For the latter one it is also possible to pick one 
interpolation filter randomly out of the family of interpolation filter. This 
introduced randomness helps to destroy periodicity in areas of moire 
1 5 artifacts. 



Upsampling Combined With Inverse Wavelet Transform 

A similar technique can be applied for image enlargement via 
interpolation using a linear filter. As for resolution reduction, the length of 
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the interpolation filter determines the scale at which the interpolation using 
wavelet coefficients may be implemented. Different interpolation filters 
might be chosen depending on characteristics of wavelet coefficients (e.g., a 
smooth filter when coefficients are small, a sharpening filter when 
5 coefficients are large, a filter with some random characteristics in halftone 
regions, etc.). 

Inverse Wavelet Transform 

A normal inverse transform is performed on the processed wavelet 
coefficients and the scaling coefficients. In one embodiment, the inverse 
1 0 transform is performed for all L levels. In an alternative embodiment, only 
L-L*^ levels are inverted and the remaining L*** levels of wavelet coefficients 
are used for the combined inverse-downsampling transform. If L=L*, no 
pure inverse is performed. 



15 Gamma Correction 

Gamma correction is not done directly on the wavelet coefficients and 
is performed in the traditional way in the pixel domain. Gamma correction 
may be performed on the scanned image, and before halftoning. 
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Halftoning 

After gamma correction, the halftoning process based on a 
wavelet-classifier may be performed. In one embodiment, dither matrices 
are selected depending on the classifier or changing error diffusion kernels 
5 depending on the classifier. 

Dithering may be used for halftoning and is commonly done by 
adding a pseudo random "noise" or "dither" signal to pixel intensities and 
then thresholding. The signal can have various property that lead to 
different halftoned images (Ordered Dither, Clustered Dot, Blue Noise 
10 Mask). 

In one embodiment, halftoning is preferred in the wavelet domain. 
For that purpose, a scale L is selected and White noise is added to the 
coefficients at scale /=1...L. The variance of the noise might differ from scale 
to scale. For example, noise variance might increase from coarse to fine 
1 5 scales. Then an inverse transform is performed and a simple thresholding is 
applied to the reconstructed image to obtain a bilevel image. The noise that 
is added to the wavelet coefficients becomes blue noise in the pixel domain. 

A prior art "blue noise mask" stores a large dither signal, for example, 
(256 X 256) because it is computational difficult to generate "colored" noise in 
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the spatial domain. However, in the wavelet domain adding noise of 
different intensities to different subbands provides an easy way to generate 
"blue noise" or any other desired "colored" noise. Thus, noise may be added 
in the wavelet domain and may be done so that different amounts of noise 
5 are added to different bands* For example, text is usually visible in HL, LH, 
so less noise is added there; more noise would be added in the HH. 



An Exemplary Copier 

Figure 7 is a block diagram of one embodiment of a digital copier. 

1 0 Referring to Figure 7, digital copier 700 comprises a scanner 701 to scan a 
hard copy document to create a scanned image. The image processing that 
contains wavelet-based processing 702 is coupled to scanner 701 to receive 
the scanned image and generate a processed image. In one embodiment, 
image processing 702 comprises the processing path shown in Figure 2C. A 

1 5 printer engine 703 is coupled to image processing to print the processed 
image. 

In one embodiment, copier 700 includes an input to receive electronic 
versions of documents that may be sent directly to image processing 702 and 
then to print engine 703. 
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An Exemplary Computer System 

An image processing path may be implemented with a computer 
system such as shown in Figure 8. Referring to Figure 8, computer system 
5 800 may comprise an exemplary client or server computer system in which 
the features described herein may be implemented. Computer system 800 
comprises a communication mechanism or bus 811 for communicating 
information, and a processor 812 coupled with bus 811 for processing 
information. Processor 812 includes a microprocessor, but is not limited to a 
1 0 microprocessor, such as Pentium™, PowerPC™, Alpha™, etc. 

System 800 further comprises a random access memory (RAM), or 
other dynamic storage device 804 (referred to as main memory) coupled to 
bus 811 for storing information and instructions to be executed by processor 
812. Main memory 804 also may be used for storing temporary variables or 
1 5 other intermediate information during execution of instructions by 
processor 812. 

Computer system 800 also comprises a read only memory (ROM) 
and/or other static storage device 806 coupled to bus 811 for storing static 
information and instructions for processor 812, and a data storage device 
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807, such as a magnetic disk or optical disk and its corresponding disk drive. 
Data storage device 807 is coupled to bus 811 for storing information and 
instructions. 

Computer system 800 may further be coupled to a display device 821, 
5 such as a cathode ray tube (CRT) or liquid crystal display (LCD), coupled to 
bus 811 for displaying information to a computer user. An alphanumeric 
^-f input device 822, including alphanumeric and other keys, may also be 

in coupled to bus 811 for communicating information and command selections 

to processor 812. An additional user input device is cursor control 823, such 
^^^^ 10 as a mouse, trackball, trackpad, stylus, or cursor direction keys, coupled to 

bus 811 for communicating direction information and command selections 
i;J to processor 812, and for controlling cursor movement on display 821. 

□ Another device that is coupled to bus 811 is hard copy device 824 

operating as a print engine, which may be used for printing images, 
1 5 instructions, data, or other information on a medium such as paper, fibn, or 
similar types of media. A scanner 830 is coupled to bus 811 to generate a 
scanned image. 



49 

Furthermore, a sound recording and playback device, such as a 
speaker and/or microphone may optionally be coupled to bus 811 for audio 
interfacing with computer system 800. 

Note that any or all of the components of system 800 and associated 
5 hardware may be used in the present invention. However, it can be 

appreciated that other configurations of the computer system may include 
some or all of the devices. 

Whereas many alterations and modifications of the present invention 
will no doubt become apparent to a person of ordinary skill in the art after 
1 0 having read the foregoing description, it is to be xmderstood that any 

particular embodiment shown and described by way of illustration is in no 
way intended to be considered limiting. Therefore, references to details of 
various embodiments are not intended to limit the scope of the claims which 
in themselves recite only those features regarded as essential to the 
1 5 invention. 



Thus, an image processing process has been described. 
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CLAIMS 

We claim: 



1 1. A system comprising: 

2 a wavelet-based image processing path to enhance an input image in 

3 a wavelet domain; and 

4 a print engine coupled to the processing path. 

1 2. The system defined in Claim 1 wherein the image processing 

2 path comprises: 

3 a forward wavelet transform; 

4 one or more wavelet-based processing blocks; and 

5 an inverse wavelet transform. 

1 3. The system defined in Claim 2 wherein the forward wavelet 

2 transform comprises a critically sampled wavelet transform. 



1 



2 



4. The system defined in Claim 2 wherein the forward wavelet 
transform comprises an overcomplete wavelet transform. 



1 



2 
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5. The system defined in Claim 2 wherein the forward wavelet 
transform comprises a Haar wavelet transform. 



1 6* A system defined in Claim 2 wherein the forward wavelet 

2 transform comprises a 5,3 wavelet transform. 

1 7, A system defined in Claim 2 wherein the forward wavelet 

2 transform comprises a 2,6 wavelet transform. 

1 8. A system defined in Claim 2 wherein the forward wavelet 

2 transform comprises a complex wavelet transform. 

1 9. A system defined in Claim 2 wherein the forward wavelet 

2 transform comprises a limited redimdancy wavelet transform. 



1 

2 
3 



10. The system defined in Claim 1 wherein the image processing 
path comprises: 

a forward wavelet transform; 
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4 a denoising processing block coupled to the forward wavelet 

5 transform to perform denoising based on at least one given threshold; and 

6 an inverse wavelet transform. 

1 11. The system defined in Claim 10 wherein the denoising 

2 processing block sets coefficients below the at least one given threshold to a 

3 predetermined value less than the threshold. 



1 12. The system defined in Claim 11 wherein the at least one given 

2 threshold is calculated based on the presence of additive Gaussian white 

3 noise in an image. 

1 13. The system defined in Claim 11 wherein the at least one given 

2 threshold is calculated from a series of test images. 

1 14. The system defined in Claim 11 wherein the at least one given 

2 threshold comprises different thresholds applied to different levels of 

3 decomposition. 



1 



2 
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15. The system defined in Claim 11 wherein the at least on given 
threshold comprises different thresholds are applied to different bands. 



1 16. The system defined in Claim 11 wherein the at least one given 

2 threshold is set manually. 

1 17. The system defined in Claim 11 wherein the at least one given 

2 threshold is set based on the standard deviation or medium of absolute 

3 values of coefficients in the region. 

1 18. The system defined in Claim 17 wherein the at least one given 

2 threshold is set on the standard deviation of HH coefficients at level one. 

1 19. The system defined in Claim 11 wherein the at least one given 

2 threshold is set in the standard deviation or medium of absolute values of 

3 HH coefficients at each level. 

1 20. The system defined in Claim 11 wherein the at least one given 

2 threshold is set based on the standard deviation or medium of absolute 
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3 values of H/HL/HH coefficients at a first level at a particular level to create 

4 a band dependent threshold. 

1 21, The system defined in Claim 11 wherein the at least one given 

2 threshold is set based on a standard deviation or medium of absolute values 

3 of LH/HL/HH coefficients at each level. 

1 22. The system defined in Claim 11 wherein the at least one given 

2 threshold is set manually using controls on a user interface. 

1 23. The system defined in Claim 11 wherein the at least one given 

2 threshold is set by computing local variances/mediums of coefficients in 

3 between a band. 

1 24. The system defined in Claim 11 wherein the at least one given 

2 threshold is set using a classifier. 
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1 25. The system defined in Claim 11 wherein the denoising 

2 processing block sets all coefficients below the at least one given threshold to 

3 zero. 



1 26. The system defined in Claim 1 wherein the image processing 

2 path comprises: 

3 a forward wavelet transform; 

4 a sharpening or smoothing processing block coupled to the forward 

5 wavelet transform to sharpen or smooth coefficients based on the magnitude 

6 of individual coefficients; and 

7 an inverse wavelet transform. 



1 27. The system defined in Claim 26 wherein the sharpening or 

2 smoothing processing block multiplies wavelet coefficients with a scale 

3 dependent factor. 



1 



2 



3 



28. The system defined in Claim 27 wherein the scale-dependent 
factor comprises: 
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4 where R is a renormalization factor and a is the parameter that determines 

5 the degree of sharpening or smoothing. 

1 29. The system defined in Claim 26 wherein the sharpening or 

2 smoothing processing block normalizes coefficients after sharpening or 

3 smoothing. 

1 30. The system defined in Claim 28 wherein the sharpening or 

2 smoothing processing block uses a scale dependent factor that is selected to 

3 include normalization. 

1 31. The system defined in Claim 1 wherein the image processing 

2 path comprises a linear interpolation filter that is applied to wavelet 

3 coefficients. 



1 32. The system defined in Claim 31 wherein the image processing 

2 path comprises: 

3 an inverse wavelet transform to inverse transform coefficients down 

4 to a predetermined level; 
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5 a downsampling block coupled to the inverse wavelet transform to 

6 downsample wavelet coefficients. 

1 33. The system defined in Claim 32 wherein the downsampling 

2 block performs either a deterministic or random downsampling based on 

3 coefficient size at the predetermined level. 

1 34. The system defined in Claim 32 wherein the downsampling 

2 block uses at least one interpolation filter based on a value of a coefficient 

3 with respect to a predetermined threshold. 

1 35. The system defined in Claim 34 wherein the at least one 

2 interpolation filter comprises a Key's filter. 

1 36. The system defined in Claim 34 wherein the downsampling 

2 processing block applies a smoothing filter to wavelet coefficients that are 

3 smaller than a first threshold and a sharpening filter to wavelet coefficients 

4 that are larger than a second threshold. 
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1 37, The system defined in Claim 1 wherein the image processing 

2 path comprises: 

3 a forward wavelet transform; 

4 a denoising processing block coupled to the forward wavelet 

5 transform to perform denoising based on a given threshold; 

6 a sharpening or smoothing processing block coupled to the denoising 

7 processing block to sharpen or smooth coefficients based on the magnitude 

8 of individual coefficients; 

9 an inverse wavelet transform coupled to the sharpening or smoothing 

1 0 processing block; and 

11 a downsampling block coupled to the inverse wavelet transform to 

1 2 downsample wavelet coefficients. 

1 38. The system defined in Claim 1 further comprising an input 

2 operable to receive the input image from an external source and a scanner 

3 for generating the input image, wherein the input and the scanner are 

4 coupled to the image processing path. 



1 



39. A method comprising: 
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2 processing an input image by enhancing the input image, including 

3 applying a forward wavelet transorm to create a plurality of coefficients and 

4 filtering coefficients with a coefficient domain operator in a wavelet domain; 

5 and 

6 outputting a processed image. 

1 40. The method defined in Claim 39 further comprising: 

2 applying one or more wavelet-based processing blocks to coefficients 

3 resulting from applying the forward wavelet transform; and 

4 applying an inverse wavelet transform. 

1 41. The method defined in Claim 40 wherein the forward wavelet 

2 transform comprises a critically sampled wavelet transform. 

1 42. The method defined in Claim 40 wherein the forward wavelet 

2 transform comprises an overcomplete wavelet transform. 

1 43. The method defined in Claim 40 wherein the forward wavelet 

2 transform comprises a Haar wavelet transform. 



1 



2 
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44. A system defined in Claim 40 wherein the forward wavelet 
transform comprises a 5,3 wavelet transform. 



1 45. A system defined in Claim 40 wherein the forward wavelet 

2 transform comprises a 2,6 wavelet transform. 

1 46. A system defined in Claim 40 wherein the forward wavelet 

2 transform comprises a complex wavelet transform. 

1 47. A system defined in Claim 40 wherein the forward wavelet 

2 transform comprises a limited redimdancy wavelet transform. 



1 48. The method defined in Claim 39 further comprising: 

2 performing denoising on the plurality of coefficients based on a given 

3 threshold; and 

4 applying an inverse wavelet transform on the plurality of coefficients 

5 after denoising. 
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1 49. The method defined in Claim 1 wherein performing denoising 

2 comprises setting coefficients below the at least one given threshold to a 

3 predetermined value. 

1 50. The system defined in Claim 49 wherein the at least one given 

2 threshold is calculated based on the presence of additive Gaussian white 

3 noise in an image. 

1 51. The system defined in Claim 49 wherein the at least one given 

2 threshold is calculated from a series of test images. 

1 52. The system defined in Claim 49 wherein the at least one given 

2 threshold comprises different thresholds are applied to different levels of 

3 decomposition. 



1 



2 



53. The system defined in Claim 49 wherein the at least one given 
threshold comprises different thresholds are applied to different bands. 
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1 54. The system defined in Claim 49 wherein the at least one given 

2 threshold is set manually. 

1 55. The system defined in Claim 49 wherein the at least one given 



2 threshold is set based on the standard deviation or medium of absolute 

3 values of coefficients in the region. 

1 56. The system defined in Claim 55 wherein the at least one given 

2 threshold is set on the standard deviation of HH coefficients at level one. 

1 57. The system defined in Claim 49 wherein the at least one given 

2 threshold is set in the standard deviation or medium of absolute values of 

3 HH coefficients at each level. 

1 58. The system defined in Claim 49 wherein the at least one given 

2 threshold is set based on the standard deviation or medium of absolute 

3 values of H/HL/HH coefficients at a first level at a particular level to create 

4 a band dependent threshold. 
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1 59. The system defined in Claim 49 wherein the at least one given 

2 threshold is set based on a standard deviation or medium of absolute values 

3 of LH /HL /HH coefficients at each level. 

1 60. The system defined in Claim 49 wherein the at least one given 

2 threshold is set manually using controls on a user interface. 

1 61. The system defined in Claim 49 wherein the at least one given 

2 threshold is set by computing local variances/mediums of coefficients in 

3 between a band. 



1 62. The system defined in Claim 49 wherein the at least one given 

2 threshold is set using a classifier. 

1 63. The method defined in Claim 49 wherein performing 

2 denoising comprises setting all coefficients below the at least one given 

3 threshold to zero. 



1 



64.. The method defined in Claim 39 further comprising: 
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2 sharpening or smoothing coefficients based on the magnitude of 

3 individual coefficients; and 

4 applying an inverse wavelet transform on the plurality of coefficients 

5 after sharpening or smoothing. 

1 65. The method defined in Claim 64 wherein sharpening or 

2 smoothing comprises multiplying wavelet coefficients with a scale 

3 dependent factor. 



1 66. The method defined in Claim 65 wherein the scale-dependent 

2 factor comprises: 

3 ^ij=R-2"j 

4 where R is a renormalization factor and a is the parameter that determines 

5 the degree of sharpening or smoothing. 

1 67. The method defined in Claim 64 wherein sharpening or 

2 smoothing comprises normalizing coefficients. 



1 



2 
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68. The method defined in Claim 39 further comprising applying a 
linear interpolation filter to wavelet coefficients. 



1 69. The method defined in Claim 39 further comprising: 

2 applying an inverse wavelet transform to inverse transform 

3 coefficients down to a predetermined level; and 

4 downsampling wavelet coefficients. 

1 70. The method defined in Claim 69 wherein downsampling the 

2 wavelet coefficients comprises performing either a deterministic or random 

3 sampling based on coefficient size at the predetermined level. 

1 71. The method defined in Claim 69 wherein downsampling 

2 wavelet coefficients comprises using at least one interpolation filter based on 

3 a value of a coefficient with respect to a predetermined threshold. 

1 72. The method defined in Claim 71 wherein the at least one 

2 interpolation filter comprises a Key's filter. 
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1 73. The method defined in Claim 69 wherein downsampling 

2 comprises applying a smoothing filter to wavelet coefficients that are smaller 

3 than a first threshold and a sharpening filter to wavelet coefficients that are 

4 larger than a second threshold. 

1 74. The method defined in Claim 39 further comprising: 

2 performing denoising on coefficients of the plurality of coefficients 

3 based on a given threshold; 

4 sharpening or smoothing coefficients of the plurality of coefficients 

5 based on the magnitude of individual coefficients; and 

6 applying an inverse wavelet transform to the plurality of coefficients 

7 and downsampling wavelet coefficients. 

1 75. A system comprising: 

2 an image enhancement subsystem having a classifier to control one or 

3 more of the following: denoising, sharpening, smoothing, halftoning, imder 

4 color removal; and 

5 an output coupled to the image enhancement subsystem. 



1 



2 
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76. The system defined in Claim 75 wherein the image 
enhancement subsystem operates, at least in part, in the wavelet domain. 



1 77. A method comprising: 

2 computing a classifier; 

3 selecting different denoising thresholds for halftone and text regions 

4 of an image based on the classifier. 



1 78. The method defined in Claim 77 wherein computing the 

2 classifier comprises computing the classifier from wavelet coefficients. 



1 79. The method defined in Claim 78 wherein computing a 

2 classifier comprises computing a standard deviation of wavelet coefficients 

3 in different bands. 



1 



2 



3 



80. The method defined in Claim 78 wherein computing a 
classifier comprises computing a standard deviation of wavelet coefficients 
in different decomposition levels. 
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1 81. The method defined in Claim 78 wherein computing a 

2 classifier comprises computing a standard deviation of wavelet coefficients 

3 across different bands. 

1 82. The method defined in Claim 78 wherein computing a 

2 classifier comprises computing a first absolute movement. 

1 83. The method defined in Claim 78 further comprising: 

2 computing a standard deviation of wavelet coefficients; and 

3 applying different denoising and sharpening/ smoothing parameters 



4 to wavelet coefficients in text halftone and backgroimd regions of image 

5 data. 



1 84. A method comprising: 

2 applying a forward wavelet transform to image data: 

3 performing denoising by thresholding coefficients generated by 

4 applying the forward wavelet transform; 

5 rescaling coefficients by filtering coefficients after thresholding. 



1 



2 
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85. The method defined in Claim 84 further comprising sampling 
the wavelet coefficients. 



1 86. The method defined in Claim 84 further comprising applying 

2 an inverse wavelet transform on filtered coefficients. 

1 87. The method defined in Claim 84 wherein scaling coefficients 

2 comprises performing sharpening or smoothing by multiplying coefficients 

3 with a level dependent parameter. 

1 88. The method defined in Claim 87 wherein high frequency bands 

2 are multiplied with a larger factor than low frequency bands to perform 

3 sharpening. 

1 89. The method defined in Claim 87 wherein low frequency bands 

2 are multiplied by a larger factor than the high frequency bands to perform 

3 smoothing. 



1 



2 
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90. The method defined in Claim 88 wherein rescaling coefficients 
comprises filtering LL band coefficients with a high pass filter. 



1 91. The method defined in Claim 89 wherein rescaling coefficients 

2 comprises filtering LL band coefficients with a low pass filter. 

1 92. The method defined in Claim 84 further comprising filtering of 

2 a subband at a specific level decomposition to modify the subband. 

1 93. The method defined in Claim 92 further comprising filtering 

2 the LL component at the largest decomposition level with a sharpening 

3 filter. 

1 94. The method defined in Claim 92 further comprising filtering 

2 the LL component at the largest decomposition level with a smoothing filter. 



1 



2 



3 



95. The method defined in Claim 84 wherein filtering coefficients 
comprises removing periodic patterns in a specific band by applying a filter 
to the specific band. 
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1 96. The method defined in Claim 95 wherein the filter comprises a 

2 band pass filter. 

1 97. The method defined in Claim 95 wherein the filter comprises a 

2 notch filter. 

1 98. The method defined in Claim 84 wherein the period is 3. 

1 99. The method defined in Claim 84 wherein the period is 5. 

1 100. The method defined in Claim 84 further comprising 

2 performing halftoning. 

1 101. The method defined in Claim 100 wherein performing 

2 halftoning comprises adding white noise with a scaled dependent variance 

3 to wavelet coefficients. 
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1 102. The method defined in Claim 100 wherein performing 

2 halftoning comprises adding white noise with a subband dependent 

3 variance to wavelet coefficients. 

1 103. The method defined in Claim 84 further comprising 

2 resampling image data. 

1 104. The method defined in Claim 103 further comprising 

2 performing an inverse wavelet transform. 

1 105. The method defined in Claim 104 wherein performing 

2 resampling image data and performing inverse wavelet transform are 

3 performed simultaneously by performing resampling directly on wavelet 

4 coefficients. 



1 



2 



106. The method defined in Claim 105 wherein the resampling is 
performed using an interpolation filter. 
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1 107. The method defined in Claim 106 wherein the interpolation 

2 filter comprises a Key's interpolation filter which is implemented on Haar 

3 wavelet coefficients. 

1 108. The method defined in Claim 107 wherein performing 

2 resampling using an interpolation filter comprises computing an 

3 overcomplete transform and critically sampling coefficients. 

1 109. The method defined in Claim 107 wherein the Key's 

2 interpolation filter performs the following procedure on wavelet coefficients: 

3 x_int = + + a2 + ^3 )sf + {uq ^a^-a^-a^ )df' + {a^ - + - . 



1 110. A method comprising: 

2 applying a forward wavelet transform to image data; 

3 performing a filtering operation on coefficients with a coefficient 

4 domain operator; and 

5 applying an inverse wavelet transform to coefficients after filtering. 
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1 111. The method defined in Claim 110 wherein performing a 

2 filtering operation comprises filtering a subband at a specific level 

3 decomposition to modify the single subband. 

1 112. The method defined in Claim 111 wherein performing a 

2 filtering operation comprises filtering the LL component at the largest 

3 decomposition level with a sharpening filter. 

1 113. The method defined in Claim 111 wherein performing a 

2 filtering operation comprises filtering the LL component at the largest 

3 decomposition level with a smoothing filter. 

1 114. The method defined in Claim 110 wherein performing a 

2 filtering operation comprises removing periodic patterns in a specific band 

3 by applying a filter to the specific band. 



1 



2 



115. The method defined in Claim 114 wherein the filter comprises 
a band pass filter. 
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1 116. The method defined in Claim 114 wherein the filter comprises 

2 a notch filter. 

1 117. The method defined in Claim 110 wherein the period is 3. 

1 118. The method defined in Claim 110 wherein the period is 5. 

1 119. A copier having a wavelet-based image processing path for 

2 enhancing image data. 

1 120. A printer having a wavelet-based image processing path for 

2 enhancing image data. 
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ABSTRACT OF THE DISCLOSURE 

A system is described. In one embodiment, the system comprises an 
image processing path and a print engine coupled to the image processing 
path. The image processing path processes an input image in a wavelet 
domain. 
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DECLARATION AND POWER OF ATTORNEY FOR PATENT APPLICATION 



As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below, next to my name. 

I believe I am the original, first, and sole inventor (if only one name is listed below) or an original, 
first, and joint inventor (if plural names are listed below) of the subject matter which is claimed and 
for which a patent is sought on the invention entitled 

WAVELET-BASED IMAGE PROCESSING PATH 



the specification of which 

X is attached hereto. 

was filed on as 

United States Application Number 

or POT International Application Number 

and was amended on - 

(if applicable) 

I hereby state that I have reviewed and understand the contents of the above-identified 
specification, including the claim(s), as amended by any amendment referred to above. I do not 
know and do not believe that the claimed invention was ever known or used in the United States of 
America before my invention thereof, or patented or described in any printed publication in any 
country before my invention thereof or more than one year prior to this application, that the same 
was not in public use or on sale in the United States of America more than one year prior to this 
application, and that the invention has not been patented or made the subject of an inventor's 
certificate issued before the date of this application in any country foreign to the United States of 
America on an application filed by me or my legal representatives or assigns more than twelve 
months (for a utility patent application) or six months (for a design patent application) prior to this 
application. 

I acknowledge the duty to disclose all information known to me to be material to patentability as 
defined in Title 37, Code of Federal Regulations, Section 1.56. 

1 hereby claim foreign priority benefits under Title 35, United States Code, Section 119(a)-(d), of any 
foreign application(s) for patent or inventor's certificate listed below and have also identified below 
any foreign application for patent or inventor's certificate having a filing date before that of the 
application on which priority is claimed: 
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Prior Foreign Application(s) 



Priority 
Claimed 



(Number) (Country) (Day/Month/Year Filed) Yes No 



(Number) (Country) (Day/MonthA'ear Filed) Yes No 



(Number) (Country) (Day/MonthA'ear Filed) Yes No 

I hereby claim the benefit under title 35, United States Code, Section 11 9(e) of any United States 
provisional application(s) listed below: 



(Application Number) Filing Date 



(Application Number) Filing Date 

I hereby claim the benefit under Title 35, United States Code, Section 120 of any United States 
application(s) listed below and, insofar as the subject matter of each of the claims of this application 
is not disclosed in the prior United States application in the manner provided by the first paragraph 
of Title 35, United States Code, Section 1 12, 1 acknowledge the duty to disclose all information 
known to me to be material to patentability as defined in Title 37, Code of Federal Regulations, 
Section 1 .56 which became available between the filing date of the prior application and the national 
or PCT international filing date of this application: 



(Application Number) Filing Date (Status patented, 

pending, abandoned) 



(Application Number) Filing Date (Status - patented, 

pending, abandoned) 

I hereby appoint the persons listed on Appendix A hereto (which is incorporated by reference and a 
part of this document) as my respective patent attorneys and patent agents, with full power of 
substitution and revocation, to prosecute this application and to transact all business in the Patent 
and Trademark Office connected herewith. 

Send correspondence to Michael J. Mallie , BLAKELY, SOKOLOFF, TAYLOR & 

(Name of Attorney or Agent) 
ZAFMAN LLP, 12400 Wilshire Boulevard 7th Floor, Los Angeles, California 90025 and direct 

telephone calls to Michael J. Mallie , (408) 720-8300. 

(Name of Attorney or Agent) 



Rev. 06/27/00 (D1) 



-2- 



I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made 
are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United 
States Code and that such willful false statements may jeopardize the validity of the 
application or any patent issued thereon. 

Full Name of Sole/First Inventor Kathrin Berkner 

Inventor's Signature ^ Date 

Residence Citizenship 

(City, State) (Country) 

Post Office Address 



Full Name of Second/Joint Inventor Edward L. Schwartz 

Inventor's Signature ^ Date 

Residence . Citizenship 

(City, State) (Country) 

Post Office Address 



Full Name of Third/Joint Inventor Michael Gormish 

Inventor's Signature Date 

Residence Citizenship 

(City, State) (Country) 

Post Office Address 



Full Name of Fourth/Joint Inventor 

Inventor's Signature Date 

Residence Citizenship 

(City, State) (Country) 

Post Office Address ^ 
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APPENDIX A 



William E. Alford, Reg. No. 37,764; Farzad E. Amini, Reg. No. P42,261 ; Aloysius T. C. AuYeung, Reg. No. 
35,432; William Thomas Babbitt, Reg. No. 39,591; Carol F. Barry, Reg. No. 41,600; Jordan Michael 
Becker, Reg. No. 39,602; Lisa N. Benado, Reg. No. 39,995; Bradley J. Bereznak, Reg. No. 33,474; 
Michael A. Bernadicou, Reg. No. 35,934; Roger W. Blakely, Jr., Reg. No. 25,831; R. Alan Burnett, Reg. 
No. 46,149; Gregory D. Caldwell, Reg. No. 39,926; Andrew C. Chen, Reg. No. 43,544; Thomas M. 
Coester, Reg. No. 39,637; Donna Jo Coningsby, Reg. No. 41,684; Florin Corie, Reg. No. 46,244; Dennis 
M. deGuzman, Reg. No. 41,702; Stephen M. De Klerk, Reg. No. P46,503; Michael Anthony DeSanctis, 
Reg. No. 39,957; Daniel M. De Vos, Reg. No. 37,813; Robert Andrew Diehl, Reg. No. 40,992; Sanjeet 
Dutta, Reg. No. P46,145; Matthew C. Fagan, Reg. No. 37,542; Tarek N. Fahmi, Reg. No. 41,402; George 
Fountain, Reg. No. 37,374; Paramita Ghosh, Reg. No. 42,806; James Y. Go, Reg. No. 40,621; James A. 
Henry, Reg. No. 41,064; Libby N. Ho, Reg. No. P46,774; Willmore F. Holbrow III, Reg. No. P41,845; 
Sheryl Sue Holloway, Reg. No. 37,850; George W Hoover II, Reg. No. 32,992; Eric S. Hyman, Reg. No. 
30,139; William W. Kidd, Reg. No. 31,772; Sang Hui Kim, Reg. No. 40,450; Walter T. Kim, Reg. No. 
42.731; EricT. King, Reg. No. 44,188; Erica W. Kuo, Reg. No. 42,775; George Brian Leavell, Reg. No. 
45,436; Kurt P. Leyendecker, Reg. No. 42,799; Gordon R. Lindeen III, Reg. No. 33,192; Jan Carol Little, 
Reg. No. 41,181; Joseph Lutz, Reg. No. 43,765; Michael J. Mallie, Reg. No. 36,591; Andre L. Marais, 
under 37 C.F.R. § 10.9(b); Paul A. Mendonsa, Reg. No. 42,879; Clive D. Menezes, Reg. No. 45,493; 
Chun M. Ng, Reg. No. 36,878; Thien T. Nguyen, Reg. No. 43,835; Thinh V. Nguyen, Reg. No. 42,034; 
Dennis A. Nicholls, Reg. No. 42,036; Daniel E. Ovanezian, Reg. No. 41,236; Kenneth B. Paley, Reg. No. 
38,989; Marina Portnova, Reg. No. P45,750; William F. Ryann, Reg. 44,313; James H. Salter, Reg. No. 
35,668; William W. Schaal, Reg. No. 39,018; James C. Scheller, Reg. No. 31 ,195; Jeffrey Sam Smith, 
Reg. No. 39,377; Maria McCormack Sobrino, Reg. No. 31,639; Stanley W. Sokoloff, Reg. No. 25,128; 
Judith A. Szepesi, Reg. No. 39,393; Vincent P. Tassinari, Reg. No. 42,179; Edwin H. Taylor, Reg. No. 
25,129; John F. Travis, Reg. No. 43,203; Joseph A. Twarowski, Reg. No. 42,191; Tom Van Zandt, Reg. 
No. 43,219; Lester J. Vincent, Reg. No. 31,460; Glenn E. Von Tersch, Reg. No. 41,364; John 
Patrick Ward, Reg. No. 40,216; Mark L. Watson, Reg. No. P46,322; Thomas C. Webster, Reg. No. 
P46,154; Steven D. Yates, Reg. No. 42,242; and Norman Zafman, Reg. No. 26,250; my patent attorneys, 
and Firasat All, Reg. No. 45,715; and Justin M. Dillon, Reg. No. 42,486; my patent agents, of BLAKELY, 
SOKOLOFF, TAYLOR & ZAFMAN LLP, with offices located at 12400 Wilshire Boulevard, 7th Floor, 
Los Angeles, California 90025, telephone (310) 207-3800, and James R. Thein, Reg. No. 31,710, my 
patent attorney with full power of substitution and revocation, to prosecute this application and to transact 
all business in the Patent and Trademark Office connected herewith. 
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APPENDIX B 



Title 37, Code of Federal Regulations, Section 1 .56 
Duty to Disclose Information Material to Patentability 

(a) A patent by its very nature is affected with a public interest. The public interest is best served, 
and the most effective patent examination occurs when, at the time an application is being examined, the 
Office is aware of and evaluates the teachings of all information material to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in 
dealing with the Office, which includes a duty to disclose to the Office all information known to that individual 
to be material to patentability as defined in this section. The duty to disclosure information exists with respect 
to each pending claim until the claim is cancelled or withdrawn from consideration, or the application becomes 
abandoned. Information material to the patentability of a claim that is cancelled or withdrawn from 
consideration need not be submitted if the information is not material to the patentability of any claim 
remaining under consideration in the application. There is no duty to submit infonmation which is not material 
to the patentability of any existing claim. The duty to disclosure all information known to be material to 
patentability is deemed to be satisfied if all information known to be material to patentability of any claim 
issued in a patent was cited by the Office or submitted to the Office in the manner prescribed by §§1 .97(b)-(d) 
and 1 .98. However, no patent will be granted on an application in connection with which fraud on the Office 
was practiced or attempted or the duty of disclosure was violated through bad faith or intentional misconduct. 
The Office encourages applicants to carefully examine: 

(1 ) Prior art cited in search reports of a foreign patent office in a counterpart application, and 

(2) The closest information over which individuals associated with the filing or prosecution of a 
patent application believe any pending claim patentably defines, to make sure that any material information 
contained therein is disclosed to the Office. 

(b) Under this section, information is material to patentability when it is not cumulative to 
information already of record or being made or record in the application, and 

(1) It establishes, by itself or in combination with other information, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or is inconsistent with, a position the applicant takes in: 

(i) Opposing an argument of unpatentability relied on by the Office, or 

(ii) Asserting an argument of patentability. 

A prima facie case of unpatentability is established when the information compels a conclusion that a claim is 
unpatentable under the preponderance of evidence, burden-of-proof standard, giving each term in the claim 
its broadest reasonable construction consistent with the specification, and before any consideration is given to 
evidence which may be submitted in an attempt to establish a contrary conclusion of patentability. 

(c) Individuals associated with the filing or prosecution of a patent application within the 
meaning of this section are: 

(1) Each inventor named in the application; 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively involved in the preparation or prosecution of the 
application and who is associated with the inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application, 

(d) Individuals other than the attorney, agent or inventor may comply with this section by 
disclosing information to the attorney, agent, or inventor. 
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